Effect of External Memory Tools on Task Performance in a Future Command and Control Environment: A Virtual Experiment using Construct

In this paper we examine the possible benefits of increasing efficiency of external memory systems on two organizational communication structures, a legacy communication structure and a proposed future communication structure, for a logistical command and control military staff. External memory aids are a popular area of study in the field of HCI and CSCW (Norman, 1993). We first compared the organizational structures using a number of metrics from social-network and organizational modeling theory. We then conducted a virtual experiment using the Construct multi-agent organizational model to test the effect of increasingly effective external memory support on the performance of the organization in the binary-choice task. Our results show that a legacy force design obtained significantly greater performance gains from external memory aids. Essentially, the future force organizational design was more resilient under memory loss conditions. Importantly, we found that a legacy force design is more technology-dependent to produce the same performance as the proposed future force nodal structure. This is counter to the vast majority of predictions for the stafftechnology relationship of the future force. Introduction The US Army is engaged in a major effort to redesign battalion-level operations into a new organizational design known as the Unit of Action (UA). This new structure is designed to operate with high mobility and in a geographically distributed manner, in order to respond more rapidly and effectively in dynamic environments than the traditional legacy structure, characterized by centralized “one big tent” command operations. One major aspect of the UA effort is creating new collaborative and interactive visualization tools to support the new UA staff operations. For while a mobile and distributed command structure has advantages to the Army when dealing with the uncertainties of modern warfare, it also has costs, such as a reduced level of communication between previously co-located colleagues. Therefore the Army is creating new data visualization and communication technologies which seek to mitigate these costs. The visualization tools under development are intended to make the staff more effective in their new roles. One theory about what makes technology such as data visualizations a positive influence on organizational performance is that they operate as a form of external memory, that is, they effectively extend the traditional limitations of human memory. Therefore the extent to which the new data visualizations function as a form of external memory will be one critical factor in determining the success of the Unit of Action effort . This paper presents a virtual experiment which evaluates the costs and benefits to organizational performance due to differing levels of external memory support. Furthermore this paper seeks to evaluate the differential responses to such increases by the legacy force and the future force staff structures. This experiment considers the computer visualization as an external memory. In the field of Human Computer Interaction, the visualization has long been considered an external memory. (Norman, 1993) We believe that the computer, when properly linked to the human it supports, is capable of reducing many mechanisms of human forgetting (Block & Morwitz 1999). We are interested in determining the organizational performance gains for an actual organization as it receives improved computer visualizations and memory support technology. Our research question is which organization (legacy structure vs future structure) will benefit more from external memory aid tools. The legacy organization is designed to be strongly functional and hierarchical in its communication processes. The proposed future organization is designed to be task driven and flexible in its communication processes. Due to the less flexible nature of the legacy organization, we expect its performance to be more sensitive to the effects of individual memory degradation, since the legacy structure’s more static nature has less ability to compensate for decreasing individual performance. Data Collection The task, knowledge, and communication data we used as input to the virtual experiment was gathered at the US Army Ft. Lee Battle Laboratory in January 2004. We participated as observers in a simulated evaluation of the Unit of Action concept conceived and run by the Battle Lab. In this evaluation 28 officers role-played staff positions of the logistical elements of the Unit of Action structure. Their task was to command the logistical elements of the UA in providing field units with necessary provisions (e.g., food, water, fuel, ammunition, etc) during a simulated battlefield scenario. The evaluation exercise took place in a large room with laptop stations in the center where the role players sat, with large projected monitors on one wall, while observers could watch from a seating area opposite. During the exercise the participants simulated the actions they would take had they been operating in an actual combat environment. Pre-determined events were given to the role players for them to plan and respond to, for instance, new deployment orders or a helicopter crash. The players then planned, organized, and executed their response. Communication between players took place entirely through a text chat system. Players were not permitted to communicate verbally with other players outside of their functional cells while the exercise was running. The log of their group chat session was saved for later analysis. In addition, the players took a survey of our design every one to two hours, which asked them to list the ten people they communicated most with in the time since the last session (i.e., their social network), in addition to other questions mentioned below. This survey was the source of the communication network data used in our virtual experiment and mentioned later in this paper. In addition to the communication data, we also collected task and knowledge data. A task list of 56 items was developed by examining the planning documents for the experiment and by consulting with subject matter experts. The SMEs had between 15 and 22 years of experience in the Army. To reflect the player’s training, a knowledge list of 26 items was constructed using the Battlefield Operating System and Basic Branch qualifications. These represent categories of specialization within an Army battle staff. Each player was surveyed to determine which qualifications they possessed. The knowledge and task data were used as inputs to the model, explained further below. The legacy structure was constructed as a hypothetical organization consisting of the same roles as the future structure, but operating under a conventional staff strong hierarchy, where each player’s communication was restricted to only the people directly above, alongside, or below them in the hierarchy. By contrast in role-playing exercise, the players were allowed to talk to whomever they needed to based on their tasks. The record of this communication was used to create the future structure for our virtual experiment. Comparing the Organization Structures Next we used ORA (the Organizational Risk Analyzer) to compare the two organizational structures (Table 1). The purpose of ORA is to assess the level of possible organizational risk and the factors that contribute to this risk. These measures are based on work in social networks, operations research, organization theory, knowledge management, and task management. (Reminga & Carley, 2003) We found relatively large differences between the two organizations in betweenness centrality and node level, but by far the most significant difference was in the density of the two structures. Density is the number of actual connections in a network divided by the number of possible connections. In our data the proposed future structure had over 3 times the density as the legacy structure. Table 1: Network measures in the legacy and future communication structures Virtual Experiment Setup The model used was Construct (Carley, 1990; Carley & Schreiber, 2003). Construct is a multi-agent model of organizational behavior. Organizations are modeled as complex Network Measures Legacy Future % Difference node levels 2.35 2.71 15 closeness centrality 0.62 0.54 -12 betweenness centrality 0.04 0.06 31 total degree centrality 0.40 0.31 -22 information centrality 0.03 0.03 0 eigenvector centrality 0.18 0.17 -2 inverse closeness centrality 0.69 0.61 -12 efficiency 0.64 0.61 4 density 0.41 0.89 334 systems of individually acting agents. The agents are all identical rule-based actors. The rules are based on known characteristics of human psychology, although they do not attempt to capture the detail and nuance of cognitive models such as ACT-R (Anderson, 1993) or SOAR (Lewis, 1999). In Construct, there is a “global” store of knowledge known as the knowledge base which represent all the knowledge that exists in the model. Each agent has its own knowledge base which represents that agent’s best guess about the content of the global knowledge base. During each time step of the simulation, each agent interacts with other agents with the goal of improving its own representation of the true knowledge base. The agents use information seeking and relative similarity to choose which other agents to exchange information with. The performance of the organization is measured using the binary choice task, where each agent decides if there are more 1’s or 0’s in an arbitrary section of the global knowledge base, based on the local knowledge of the agent at the time. Overall performance for one time step is then the percent of correct choices made by the agents (Schreiber & Carley 2003). For this virtual experiment we used a 2x6 experimental design, with two different access structures (legacy and future), and six levels of external memory support (100% through 95% at 1% intervals). The experiment was run 100 times per cell. The communication network we gathered for the future force and the communication network we generated for the legacy force were also used as inputs to the model. In Construct, the supplied communication network functions not as the actual communication structure between the agents, but as a constraint on the possible communication that takes place. In other words, if there was a link between two actors in the provided communication structure, then it was possible, but not required, that those two agents could interact during the simulation. The knowledge and task data collected above were used as identical inputs to the model for both the legacy and future force treatments. The agent by knowledge network we gathered provided the initial knowledge string for each agent. The knowledge by task network we gathered provided the resource requirements for each task. Using real-world task, knowledge, and communication data grounded this virtual experiment more fully in the real world. To represent the effect of external memory, we used the forgetting function available in Construct. This parameter describes the percent of each agent’s memory which is lost in each time step. We varied the degree of external memory support by changing the handicap imposed on each individual’s memory. For instance, we simulated perfect external memory aids by giving all agents perfect memory. We then decremented the perfect external memory aid by increments of 1%.(Table 2) Table 2. Percent of individual forgetting based on external memory aid factor The dependent variable used in this experiment is performance on the binary choice task. The performance for each cell was determined by averaging the results of 100 runs with 100 time steps each. The communication style was one-to-one, and referral between agents was allowed (Schreiber & Carley 2003). Virtual Experiment Results Figure 1 and Table 3 present the performance of the system on the last time step for each treatment of the experiment. Each data point is the average of 100 complete runs of the model. The future structure outperformed the legacy structure at all levels of external memory aid. The differences between the structures were statistically significant using the student’s T test at the t < .0001 level for all EMA factors. Both structures showed increased performance as the forgetting was reduced (as the external memory aid got better). However the legacy structure improved more rapidly as forgetting reduced, narrowing the difference between the treatments (Figure 1). From 5 percent forgetting to 0 percent forgetting, the legacy organization performance on the binary choice task went from 33.3 to 91.5. The future organization performance went from 51.0 to 94.8. The variance of the results decreased notably from an average of 10.45 for both structures to 1.82 as forgetting was increased to zero (Table 3). This is due to the increased randomness introduced by the forgetting. As the forgetting occurs with a randomly distributed probability, an increase in forgetting increases the variance of organizational performance. Future Structure Legacy Structure t-test EMA 5 51.0 +10.1 33.3 +10.8 t(99) = -11.20, p < .0001 EMA 4 67.3 +-11.0 49.7 +10.6 t(99) = -11.05, p < .0001 EMA 3 69.3 +3.34 64.0 +4.73 t(99) = -8.67, p < .0001 EMA 2 83.2 +5.63 77.8 +4.08 t(99) = -7.72, p < .0001 EMA 1 90.2 +3.13 87.9 +3.81 t(99) = -4.59, p < .0001 EMA 0 94.8 +1.31 91.5 +2.34 t(99) = -13.17, p < .0001 External Memory Aid (EMA)